% -*- Mode: latex; -*-
% $HeadURL$
% $Id$

This chapter describes the mechanics of using \hpcrun{} and \hpclink{}
to profile an application and collect performance data.  For advice on
how to choose events, perform scaling studies, etc., see
Chapter~\ref{chpt:effective-performance-analysis} {\it Effective
Strategies for Analyzing Program Performance}.

\section{Using \hpcrun{}}

The \hpcrun{} launch script is used to run an application and collect
performance data for {\it dynamically linked\/} binaries.  For
dynamically linked programs, this requires no change to the program
source and no change to the build procedure.  You should build your
application natively at full optimization.  \hpcrun{} inserts its
profiling code into the application at runtime via \verb|LD_PRELOAD|.

The basic options for \hpcrun{} are \verb|-e| (or \verb|--event|) to
specify a sampling source and rate and \verb|-t| (or \verb|--trace|) to
turn on tracing.  Sample sources are specified as `\verb|event@howoften|'
where \verb|event| is the name of the source and \verb|howoften| is either
a number specifying the period (threshold) for that event, or \verb|f| followed by a number, \eg{}, \verb|@f100| 
specifying a target sampling frequency for the event in samples/second.\footnote{Frequency-based sampling and 
the frequency-based notation for {\tt howoften} is only
available for sample sources managed by Linux {\tt perf\_events}. For Linux {\tt perf\_events}, \HPCToolkit{} uses 
a default sampling frequency of 300 samples/second.}
Note that a higher period implies a lower rate of sampling.  
The \verb|-e| option may be used multiple times to specify that multiple
sample sources be used for measuring an execution.  
The basic syntax for profiling an application with
\hpcrun{} is:

\begin{quote}
\begin{verbatim}
hpcrun -t -e event@howoften ... app arg ...
\end{verbatim}
\end{quote}

For example, to profile an application using hardware counter sample sources
provided by Linux \verb|perf_events| and sample cycles at 300 times/second (the default sampling frequency) and sample every 4,000,000 instructions, 
you would use:

\begin{quote}
\begin{verbatim}
hpcrun -e CYCLES -e INSTRUCTIONS@4000000 app arg ...
\end{verbatim}
\end{quote}

The units for timer-based sample sources (\verb|CPUTIME|, \verb|REALTIME|, and \verb|WALLCLOCK|) are microseconds,
so to sample an application with tracing every 5,000 microseconds
(200~times/second), you would use:

\begin{quote}
\begin{verbatim}
hpcrun -t -e CPUTIME@5000 app arg ...
\end{verbatim}
\end{quote}

\hpcrun{} stores its raw performance data in a {\it measurements}
directory with the program name in the directory name.  On systems
with a batch job scheduler (eg, PBS) the name of the job is appended
to the directory name.

\begin{quote}
\begin{verbatim}
hpctoolkit-app-measurements[-jobid]
\end{verbatim}
\end{quote}

It is best to use a different measurements directory for each run.
So, if you're using \hpcrun{} on a local workstation without a job
launcher, you can use the `\verb|-o dirname|' option to specify an
alternate directory name.

For programs that use their own launch script (eg, \verb|mpirun| or
\verb|mpiexec| for MPI), put the application's run script on the
outside (first) and \hpcrun{} on the inside (second) on the command
line.  For example,

\begin{quote}
\begin{verbatim}
mpirun -n 4 hpcrun -e CYCLES mpiapp arg ...
\end{verbatim}
\end{quote}

Note that \hpcrun{} is intended for profiling dynamically linked {\it
binaries}.  It will not work well if used to launch a shell script.
At best, you would be profiling the shell interpreter, not the script
commands, and sometimes this will fail outright.

It is possible to use \hpcrun{} to launch a statically linked binary,
but there are two problems with this.  First, it is still necessary to
build the binary with \hpclink{}.  Second, static binaries are
commonly used on parallel clusters that require running the binary
directly and do not accept a launch script.  However, if your system
allows it, and if the binary was produced with \hpclink, then
\hpcrun{} will set the correct environment variables for profiling
statically or dynamically linked binaries.  All that \hpcrun{} really
does is set some environment variables (including \verb|LD_PRELOAD|)
and \verb|exec| the binary.

% ===========================================================================

\section{Using \hpclink{}}

For now, see Chapter~\ref{chpt:statically-linked-apps} on {\it
Monitoring Statically Linked Applications}.

% ===========================================================================

\section{Harware Counter Event Names}

HPCToolkit uses libpfm4\cite{libpfm-www} to translate from an event name string to an event code recognized by the kernel. 
An event name is case insensitive and is defined as followed:
\begin{verbatim}
[pmu::][event_name][:unit_mask][:modifier|:modifier=val] 
\end{verbatim}

\begin{itemize}
	\item \textbf{pmu}.  Optional name of the PMU (group of events) to which the event belongs to. This is useful to disambiguate events in case events from difference sources have the same name. If no pmu is specified, the first match event is used.
	\item \textbf{event\_name}.  The name of the event. It must be the complete name, partial matches are not accepted. 
	\item \textbf{unit\_mask}.  This designate an optional sub-events. Some events can be refined using sub-events. An event may have multiple unit masks and it is possible to combine them (for some events) by repeating \texttt{:unit\_mask} pattern.
	\item \textbf{modifier}.  A modifier is an optional filter that restricts when an event counts.
                The form of a  modifier may be either \texttt{:modifier} or \texttt{:modifier=val}.
		For modifiers without a value, the presence of the modifier is 
                interpreted as a restriction. Events may allow use of multiple modifiers 
                at the same time.
		\begin{itemize}
		\item \textbf{hardware event modifiers}. Some hardware events support one or more modifiers that restrict counting to a subset of events. For instance, on an Intel Broadwell EP, one can add a modifier to \verb|MEM_LOAD_UOPS_RETIRED| to count only load operations that are 
an \verb|L2_HIT| or an \verb|L2_MISS|. For information about all modifers for hardware events, 
one can direct \HPCToolkit{}'s measurement subsystem to list all native events and their modifiers 
as described in Section~\ref{sample-sources}. 
		\item \textbf{precise\_ip}. For some events, it is possible to control the amount of skid. 
Skid is a measure of how many instructions may execute between an event and the PC where the event is reported. 
Smaller skid enables more accurate attribution of events to instructions. Without a skid modifier, hpcrun allows arbitrary skid because some architectures 
don't support anything more precise.  One may optionally specify one of the following as a skid modifier:
			\begin{itemize}
				\item \verb|:p| : a sample must have constant skid.
				\item \verb|:pp| :  a sample is requested to have 0 skid.
				\item \verb|:ppp| : a sample must have 0 skid.  
				\item \verb|:P| : autodetect the least skid possible.
			\end{itemize}
			NOTE: If the kernel or the hardware does not support the specified value of the skid, no error message will be reported 
but no samples will be delivered.
	\end{itemize}
\end{itemize}


% ===========================================================================

\section{Sample Sources}
\label{sample-sources}

This section provide an overview of how to use sample sources supported by HPCToolkit.  To
see a list of the available sample sources and events that \hpcrun{}
supports, use `\verb|hpcrun -L|' (dynamic) or set
`\verb|HPCRUN_EVENT_LIST=LIST|' (static).  Note that on systems with
separate compute nodes, it is best to run this on a compute node.

\newcommand{\perfevents}{{\tt perf\_events}}

\subsection{Linux \perfevents}

Linux \perfevents{} provides a powerful interface that supports 
measurement of both application execution and kernel activity. 
Using
\perfevents{}, one can measure both hardware and software events. 
Using a processor's hardware performance monitoring unit (PMU), the
\perfevents{} interface can measure an execution using any hardware counter
supported by the PMU. Examples of hardware events include cycles, instructions
completed, cache misses, and stall cycles. Using instrumentation built in to the Linux kernel,
the \perfevents{} interface can measure software events. Examples of software events include page
faults, context switches, and CPU migrations. 



\subsubsection{Capabilities of HPCToolkit's \perfevents{} Interface}

\paragraph{Frequency-based sampling.} 
The Linux \perfevents{} interface supports frequency-based sampling. 
With frequency-based sampling, the kernel automatically selects and adjusts an event period with the
aim of delivering samples for that event at a target sampling frequency.\footnote{The 
kernel may be unable to deliver the desired frequency if 
there are fewer events per second than the desired frequency.}  
Unless a user explicitly specifies an event count threshold for an event, 
HPCToolkit's measurement interface will use frequency-based sampling by default.  
HPCToolkit's default sampling frequency is ${\rm min}(300,M-1)$, where $M$ is the 
value specified in the system configuration file \verb|/proc/sys/kernel/perf_event_max_sample_rate|.

For circumstances where the user wants to use frequency-based sampling but 
HPCToolkit's default sampling frequency is inappropriate, 
one can specify the target sampling frequency for a particular event using the notation 
{\em event}{\tt @f}{\em rate} when specifying an event or change the default sampling frequency.
When measuring a dynamically-linked executable using {\tt hpcrun}, one can change the default sampling frequency using {\tt hpcrun}'s {\tt -c} option. To set a new default sampling frequency for a statically-linked executable instrumented with {\tt hpclink}, set the \verb|HPCRUN_PERF_COUNT| environment variable.
The section below entitled {\em Launching} provides  
examples of how to monitor an execution using frequency-based sampling.

\paragraph{Multiplexing.} 
Using multiplexing enables one to monitor more events
in a single execution than the number of hardware counters a processor
can support for each thread. The number of events that can be monitored in
a single execution is only limited by the maximum number of concurrent
events that the kernel will allow a user to multiplex using the
\perfevents{} interface.

When more events are specified than can be monitored simultaneously
using a thread's hardware counters,\footnote{How many events can be
  monitored simultaneously on a particular processor may depend on the
  events specified.} the kernel will employ multiplexing and divide
the set of events to be monitored into groups, monitor only one group
of events at a time, and cycle repeatedly through the groups
as a program executes. 

For applications that have very regular,
steady state behavior, e.g., an iterative code with lots of
iterations, multiplexing will yield results that are suitably representative
of execution behavior.  However, for executions that consist of
unique short phases, measurements collected using multiplexing may
not accurately represent the execution behavior. To obtain
more accurate measurements, one can run an application multiple times and in 
each run collect a subset of events that can be measured without multiplexing.
Results from several such executions can be imported into HPCToolkit's \hpcviewer{} 
and analyzed together.


\paragraph{Thread blocking.} When a program executes, 
a thread may block waiting for the kernel to complete some operation on its behalf.
For instance, a thread may block waiting for data to become available so that a {\tt read} operation 
can complete. On systems running Linux 4.3 or newer, one can use the \perfevents{} sample source to monitor how much time a thread is blocked and where the blocking occurs. To measure
the time a thread spends blocked, one can profile with \verb|BLOCKTIME| event and
another time-based event, such as \verb|CYCLES|. The \verb|BLOCKTIME| event shouldn't have any frequency or period specified, whereas \verb|CYCLES| should have a frequency or period specified.

\subsubsection{Launching}
\label{sec:perf-launching}

When sampling with native events, by default hpcrun will profile using \perfevents{}. 
To force HPCToolkit to use PAPI rather than \perfevents{} to oversee monitoring of a PMU event 
(assuming that HPCToolkit has been configured to include support for PAPI),
one must prefix the event with \lq{\verb|papi::|}\rq{} as follows:

\begin{quote}
\begin{verbatim}
hpcrun -e papi::CYCLES
\end{verbatim}
\end{quote}

For PAPI presets, there is no need to prefix the event with
 \lq{\verb|papi::|}\rq. For instance it is sufficient to specify \verb|PAPI_TOT_CYC| event
without any prefix to profile using PAPI.

To sample an execution 100 times per second (frequency-based sampling) counting \verb|CYCLES| 
and 100 times a second counting \verb|INSTRUCTIONS|:

\begin{quote}
\begin{verbatim}
hpcrun -e CYCLES@f100 -e INSTRUCTIONS@f100 ...
\end{verbatim}
\end{quote}

To sample an execution every 1,000,000 cycles and every 1,000,000 instructions using period-based sampling:

\begin{quote}
\begin{verbatim}
hpcrun -e CYCLES@1000000 -e INSTRUCTIONS@1000000
\end{verbatim}
\end{quote}

By default, hpcrun uses frequency-based sampling with the rate
300 samples per second per event type. Hence the following command causes HPCToolkit to
sample \verb|CYCLES| at 300 samples per second and \verb|INSTRUCTIONS| at 300 samples per second:

\begin{quote}
\begin{verbatim}
hpcrun -e CYCLES -e INSTRUCTIONS …
\end{verbatim}
\end{quote}

One can specify a different default sampling period or frequency using the \verb|-c| option. 
The command below will sample \verb|CYCLES| at 200 samples per second and \verb|INSTRUCTIONS| at 200 samples per second:

\begin{quote}
\begin{verbatim}
hpcrun -c f200 -e CYCLES -e INSTRUCTIONS …
\end{verbatim}
\end{quote}



\subsubsection{Notes}

\begin{itemize}
\item
Linux \perfevents{} uses one file descriptor for each event to be monitored. 
Furthermore, since \hpcrun{} generates one hpcrun file for each thread, and an additional hpctrace file if traces is enabled.
Hence for $e$ events and $t$ threads, the required number of file descriptors is: 
\begin{quote}
  $t \times e + t + t$ (if trace is enabled) 
\end{quote}
For instance, if one profiles a multi-threaded program that executes with 500 threads using 4 events, 
then the required number of file descriptors is
\begin{quote}
  500 threads $\times$ 4 events + 500 hpcrun files + 500 hpctrace files \\
  = 3000 file descriptors
\end{quote}
If the number of file descriptors exceeds the number of maximum opened files, then the program will crash. 
To remedy this issue, one needs to increase the number of maximum opened files.

\item
\sloppy
When a system is configured with suitable permissions, HPCToolkit will sample call stacks
within the Linux kernel in addition to application-level call stacks. This feature can be useful to measure kernel activity on behalf of a thread (e.g., zero-filling allocated pages when they are first touched) 
or to observe where, why, and how long a thread blocks.
For a user to be able to sample kernel call stacks, the configuration file
\verb|/proc/sys/kernel/perf_event_paranoid| must have a value $\leq 1$.  To associate addresses
in kernel call paths with function names, the value of
\verb|/proc/sys/kernel/kptr_restrict| must be 0 (number zero). If these settings are not configured in this way on your system, you will need someone with administrator privileges to change them for you to 
be able to sample call stacks within the kernel.

\item
Due to a limitation present in all Linux kernel versions currently available,  
HPCToolkit's measurement subsystem can only approximate a thread's blocking time.
At present, Linux reports when a thread blocks but does not report when a thread resumes execution.
For that reason, HPCToolkit's measurement subsystem approximates the time a thread spends blocked using sampling as the time between when the thread blocks and when the thread receives its first sample
after resuming execution.


\item
Users need to be cautious when considering measured counts of events that have been collected using 
hardware counter multiplexing.  Currently, it is not obvious to a user
if a metric was measured using a multiplexed counter. Information about whether a counter was 
multiplexed is only available in the \verb|experiment.xml| file produced when post-processing measurement data with \hpcprof{} or \hpcprofmpi, but is not visible in \hpcviewer.
\end{itemize}

\subsection{PAPI}

PAPI, the Performance API, is a library for providing access to the
hardware performance counters. PAPI aims to provide a
consistent, high-level interface that consists of a universal set of event names that can be used 
to measure performance on any processor, independent of any processor-specific event names. 
In some cases, PAPI event names
represent quantities synthesized by combining measurements based on multiple native events 
available on a particular processor.
For instance, in some cases PAPI reports
total cache misses by measuring and combining data misses and instruction misses. 
PAPI is available from the University of Tennessee at:

\begin{quote}
\url{http://icl.cs.utk.edu/papi/}
\end{quote}

PAPI focuses mostly on in-core CPU events: cycles, cache misses,
floating point operations, mispredicted branches, etc.  For example,
the following command samples total cycles and L2 cache misses.

\begin{quote}
\begin{verbatim}
hpcrun -e PAPI_TOT_CYC@15000000 -e PAPI_L2_TCM@400000 app arg ...
\end{verbatim}
\end{quote}

The precise set of PAPI preset and native events is highly system
dependent.  Commonly, there are events for machine cycles, cache
misses, floating point operations and other more system specific
events.  However, there are restrictions both on how many events can
be sampled at one time and on what events may be sampled together and
both restrictions are system dependent.  Table~\ref{tab:papi-events}
contains a list of commonly available PAPI events.

To see what PAPI events are available on your system, use the
\verb|papi_avail| command from the \verb|bin| directory in your PAPI
installation.  The event must be both available and not derived to be
usable for sampling.  The command \verb|papi_native_avail| displays
the machine's native events.  Note that on systems with separate
compute nodes, you normally need to run \verb|papi_avail| on one of
the compute nodes.

\begin{table}
\begin{center}
\begin{tabular}{|l|l|}
\hline
\verb|PAPI_BR_INS| & Branch instructions \\
\hline
\verb|PAPI_BR_MSP| & Conditional branch instructions mispredicted \\
\hline
\verb|PAPI_FP_INS| & Floating point instructions \\
\hline
\verb|PAPI_FP_OPS| & Floating point operations \\
\hline
\verb|PAPI_L1_DCA| & Level 1 data cache accesses \\
\hline
\verb|PAPI_L1_DCM| & Level 1 data cache misses \\
\hline
\verb|PAPI_L1_ICH| & Level 1 instruction cache hits \\
\hline
\verb|PAPI_L1_ICM| & Level 1 instruction cache misses \\
\hline
\verb|PAPI_L2_DCA| & Level 2 data cache accesses \\
\hline
\verb|PAPI_L2_ICM| & Level 2 instruction cache misses \\
\hline
\verb|PAPI_L2_TCM| & Level 2 cache misses \\
\hline
\verb|PAPI_LD_INS| & Load instructions \\
\hline
\verb|PAPI_SR_INS| & Store instructions \\
\hline
\verb|PAPI_TLB_DM| & Data translation lookaside buffer misses \\
\hline
\verb|PAPI_TOT_CYC| & Total cycles \\
\hline
\verb|PAPI_TOT_IIS| & Instructions issued \\
\hline
\verb|PAPI_TOT_INS| & Instructions completed \\
\hline
\end{tabular}
\end{center}
\caption{Some commonly available PAPI events.
The exact set of available events is system dependent.}
\label{tab:papi-events}
\end{table}

When selecting the period for PAPI events, aim for a rate of
approximately a few hundred samples per second.  So, roughly several
million or tens of million for total cycles or a few hundred thousand
for cache misses.  PAPI and \hpcrun{} will tolerate sampling rates as
high as 1,000 or even 10,000 samples per second (or more).  However, rates
higher than a few hundred samples per second will only increase measurement
overhead and distort the execution of your program; they won't yield more
accurate results.

Beginning with Linux kernel version 2.6.32,
support for accessing the performance counters is now 
built in to the standard Linux kernel.  On kernels
2.6.32 or later, PAPI can be compiled and run entirely in user mode
without patching the kernel. 

On Blue Gene platforms that are not based on Linux,
PAPI is highly recommended as it provides an essential substrate for accessing
hardware performance counters. On modern Linux systems that include support
for \verb|perf_events|, PAPI is only recommended for monitoring 
events outside the scope of the \verb|perf_events| interface. 

\paragraph{Proxy Sampling}

\HPCToolkit{} supports proxy sampling for derived PAPI events.
For \HPCToolkit{} to sample a PAPI event directly, the event must not be
derived and must trigger hardware interrupts when a threshold is exceeded.  
For events that cannot trigger interrupts directly, HPCToolkit's proxy sampling
sample on another event that is supported directly and then reads the 
counter for the derived event. In this case, 
a native event can serve as a proxy for one or more derived events.

To use proxy sampling, specify the \hpcrun{} command line as usual and
be sure to include at least one non-derived PAPI event.  The derived
events will be accumulated automatically when processing a sample trigger for a native event.
We recommend adding \verb|PAPI_TOT_CYC| as a native event when using proxy sampling, but
proxy sampling will gather data  as long as the event set contains at least one
non-derived PAPI event.  Proxy sampling requires one non-derived PAPI event to serve as the proxy;
a Linux timer can't serve as the proxy for a PAPI derived event.

For example, on newer Intel CPUs, often PAPI floating point events are
all derived and cannot be sampled directly.  In that case, you could
count FLOPs by using cycles a proxy event with a command line such as
the following.  The period for derived events is ignored and may be
omitted.

\begin{quote}
\begin{verbatim}
hpcrun -e PAPI_TOT_CYC@6000000 -e PAPI_FP_OPS app arg ...
\end{verbatim}
\end{quote}

Attribution of proxy samples is not as accurate as regular samples.
The problem, of course, is that the event that triggered the sample
may not be related to the derived counter.  The total count of events
should be accurate, but their location at the leaves in the Calling
Context tree may not be very accurate.  However, the higher up the
CCT, the more accurate the attribution becomes.  For example, suppose
you profile a loop of mixed integer and floating point operations and
sample on \verb|PAPI_TOT_CYC| directly and count \verb|PAPI_FP_OPS|
via proxy sampling.  The attribution of flops to individual statements
within the loop is likely to be off.  But as long as the loop is long
enough, the count for the loop as a whole (and up the tree) should be
accurate.

\subsection{WALLCLOCK, REALTIME and CPUTIME}

\HPCToolkit{} supports three timer-based sample sources: \verb|CPUTIME|,
\verb|REALTIME| and \verb|WALLCLOCK|.
The unit for periods of these timers is microseconds.

Before describing this capability further, it is worth noting
that the CYCLES event supported by Linux \perfevents{} or PAPI's \verb|PAPI_TOT_CYC| 
are generally superior to any of the timer-based sampling sources. 

\sloppy
The \verb|CPUTIME| and \verb|REALTIME| sample sources are based on the POSIX
timers \verb|CLOCK_THREAD_CPUTIME_ID| and \verb|CLOCK_REALTIME| with 
the Linux \verb|SIGEV_THREAD_ID| extension.  
\verb|CPUTIME| only counts time when the CPU is running;  
\verb|REALTIME| counts
real (wall clock) time, whether the process is running or not. 
Signal delivery for these timers is thread-specific, so these timers are suitable for
profiling multithreaded programs.
Sampling using the \verb|REALTIME| sample source
may break some applications that don't handle interrupted syscalls well.  In that
case, consider using \verb|CPUTIME| instead.
It is worth noting that \verb|REALTIME| and \verb|CPUTIME| are not available on Blue Gene,
where compute nodes run a custom microkernel instead of Linux.

The \verb|WALLCLOCK| sample source is based on the \verb|ITIMER_PROF|
interval timer. 
\verb|WALLCLOCK| counts time when an application is running or the kernel is running on behalf of the
application. 
Unlike the POSIX timers that support  \verb|CPUTIME| and \verb|REALTIME|,
the interval timer supporting \verb|WALLCLOCK| does not support thread-specific signal delivery 
on Linux. 
\verb|WALLCLOCK| is available on Blue Gene even though \verb|REALTIME| and \verb|CPUTIME| are not.
On Linux the maximum interrupt rate for the interval itimer is limited by the
system's Hz rate, commonly 1,000 cycles per second, but may be lower.
That is, \verb|WALLCLOCK@10| will not generate any higher sampling rate
than \verb|WALLCLOCK@1000|.  
On IBM Blue Gene, the interval timer is not
bound by the Hz rate and so sampling rates faster than 1,000 per
second are possible.

The following example, which specifies a period of 5000 microseconds will sample
each thread in \verb|app| at a rate of approximately 200 times per second.

\begin{quote}
\begin{verbatim}
hpcrun -e REALTIME@5000 app arg ...
\end{verbatim}
\end{quote}


\noindent {\it Note:} do not use more than one timer-based sample source to monitor a program execution.
When using a sample source such as \verb|CPUTIME|, \verb|REALTIME|, or \verb|WALLCLOCK|,
we recommend not using another time-based sampling source such as 
Linux \perfevents{} CYCLES or PAPI's \verb|PAPI_TOT_CYC|. 
Technically, this is feasible and \hpcrun{} won't die.
However, multiple time-based sample sources would compete with one another to measure the
execution and likely lead to dropped samples and possibly distorted results.

\subsection{IO}

The \verb|IO| sample source counts the number of bytes read and
written.  This displays two metrics in the viewer: ``IO Bytes Read''
and ``IO Bytes Written.''  The \verb|IO| source is a synchronous sample
source.  
It overrides the functions \verb|read|, \verb|write|, \verb|fread|
and \verb|fwrite| and records the number of bytes read or
written along with their dynamic context synchronously rather 
than relying on data collection triggered by interrupts.

To include this source, use the \verb|IO| event (no period).  In the
static case, two steps are needed.  Use the \verb|--io| option for
\hpclink{} to link in the \verb|IO| library and use the \verb|IO| event
to activate the \verb|IO| source at runtime.  For example,

\begin{quote}
\begin{tabular}{@{}cl}
(dynamic) & \verb|hpcrun -e IO app arg ...| \\
(static)  & \verb|hpclink --io gcc -g -O -static -o app file.c ...| \\
& \verb|export HPCRUN_EVENT_LIST=IO| \\
& \verb|app arg ...|
\end{tabular}
\end{quote}

The \verb|IO| source is mainly used to find where your program reads or
writes large amounts of data.  However, it is also useful for tracing
a program that spends much time in \verb|read| and \verb|write|.  The
hardware performance counters (PAPI) do not advance while running in
the kernel, so the trace viewer may misrepresent the amount of time
spent in syscalls such as \verb|read| and \verb|write|.  By adding the
\verb|IO| source, \hpcrun{} overrides \verb|read| and \verb|write| and
thus is able to more accurately count the time spent in these
functions.

\subsection{MEMLEAK}

The \verb|MEMLEAK| sample source counts the number of bytes allocated
and freed.  Like \verb|IO|, \verb|MEMLEAK| is a synchronous sample
source and does not generate asynchronous interrupts.  Instead, it
overrides the malloc family of functions (\verb|malloc|, \verb|calloc|,
\verb|realloc| and \verb|free| plus \verb|memalign|, \verb|posix_memalign|
and \verb|valloc|) and records the number of bytes
allocated and freed along with their dynamic context.

\verb|MEMLEAK| allows you to find locations in your program that
allocate memory that is never freed.  But note that failure to free a
memory location does not necessarily imply that location has leaked
(missing a pointer to the memory).  It is common for programs to
allocate memory that is used throughout the lifetime of the process
and not explicitly free it.

To include this source, use the \verb|MEMLEAK| event (no period).
Again, two steps are needed in the static case.  Use the \verb|--memleak|
option for \hpclink{} to link in the \verb|MEMLEAK| library
and use the \verb|MEMLEAK| event to activate it at runtime.  For
example,

\begin{quote}
\begin{tabular}{@{}cl}
(dynamic) & \verb|hpcrun -e MEMLEAK app arg ...| \\
(static)  & \verb|hpclink --memleak gcc -g -O -static -o app file.c ...| \\
& \verb|export HPCRUN_EVENT_LIST=MEMLEAK| \\
& \verb|app arg ...|
\end{tabular}
\end{quote}

If a program allocates and frees many small regions, the \verb|MEMLEAK|
source may result in a high overhead.  In this case, you may reduce
the overhead by using the memleak probability option to record only a
fraction of the mallocs.  For example, to monitor 10\% of the mallocs,
use:

\begin{quote}
\begin{tabular}{@{}cl}
(dynamic) & \verb|hpcrun -e MEMLEAK --memleak-prob 0.10 app arg ...| \\
(static)  & \verb|export HPCRUN_EVENT_LIST=MEMLEAK| \\
& \verb|export HPCRUN_MEMLEAK_PROB=0.10| \\
& \verb|app arg ...|
\end{tabular}
\end{quote}

It might appear that if you monitor only 10\% of the program's
mallocs, then you would have only a 10\% chance of finding the leak.
But if a program leaks memory, then it's likely that it does so many
times, all from the same source location.  And you only have to find
that location once.  So, this option can be a useful tool if the
overhead of recording all mallocs is prohibitive.

Rarely, for some programs with complicated memory usage patterns, the
\verb|MEMLEAK| source can interfere with the application's memory
allocation causing the program to segfault.  If this happens, use the
\verb|hpcrun| debug ({\tt dd}) variable \verb|MEMLEAK_NO_HEADER| as a
workaround.

\begin{quote}
\begin{tabular}{@{}cl}
(dynamic) & \verb|hpcrun -e MEMLEAK -dd MEMLEAK_NO_HEADER app arg ...| \\
(static)  & \verb|export HPCRUN_EVENT_LIST=MEMLEAK| \\
& \verb|export HPCRUN_DEBUG_FLAGS=MEMLEAK_NO_HEADER| \\
& \verb|app arg ...|
\end{tabular}
\end{quote}

The \verb|MEMLEAK| source works by attaching a header or a footer to
the application's \verb|malloc|'d regions.  Headers are faster but
have a greater potential for interfering with an application.  Footers
have higher overhead (require an external lookup) but have almost no
chance of interfering with an application.  The
\verb|MEMLEAK_NO_HEADER| variable disables headers and uses only
footers.

% ===========================================================================

\section{Process Fraction}

Although \hpcrun{} can profile parallel jobs with thousands or tens of
thousands of processes, there are two scaling problems that become
prohibitive beyond a few thousand cores.  First, \hpcrun{} writes the
measurement data for all of the processes into a single directory.
This results in one file per process plus one file per thread (two
files per thread if using tracing).  Unix file systems are not
equipped to handle directories with many tens or hundreds of thousands
of files.  Second, the sheer volume of data can overwhelm the viewer
when the size of the database far exceeds the amount of memory on the
machine.

The solution is to sample only a fraction of the processes.  That is,
you can run an application on many thousands of cores but record data
for only a few hundred processes.  The other processes run the
application but do not record any measurement data.  This is what the
process fraction option (\verb|-f| or \verb|--process-fraction|) does.
For example, to monitor 10\% of the processes, use:

\begin{quote}
\begin{tabular}{@{}cl}
(dynamic) & \verb|hpcrun -f 0.10 -e event@howoften app arg ...| \\
(dynamic) & \verb|hpcrun -f 1/10 -e event@howoften app arg ...| \\
(static)  & \verb|export HPCRUN_EVENT_LIST='event@howoften'| \\
& \verb|export HPCRUN_PROCESS_FRACTION=0.10| \\
& \verb|app arg ...|
\end{tabular}
\end{quote}

With this option, each process generates a random number and records
its measurement data with the given probability.  The process fraction
(probability) may be written as a decimal number (0.10) or as a
fraction (1/10) between 0 and 1.  So, in the above example, all three
cases would record data for approximately 10\% of the processes.  Aim
for a number of processes in the hundreds.

% ===========================================================================

\section{Starting and Stopping Sampling}

\HPCToolkit{} supports an API for the application to start and stop
sampling.  This is useful if you want to profile only a subset of a
program and ignore the rest.  The API supports the following
functions.

\begin{quote}
\begin{verbatim}
void hpctoolkit_sampling_start(void);
void hpctoolkit_sampling_stop(void);
\end{verbatim}
\end{quote}

For example, suppose that your program has three major phases: it
reads input from a file, performs some numerical computation on the
data and then writes the output to another file.  And suppose that you
want to profile only the compute phase and skip the read and write
phases.  In that case, you could stop sampling at the beginning of the
program, restart it before the compute phase and stop it again at the
end of the compute phase.

This interface is process wide, not thread specific.  That is, it
affects all threads of a process.  Note that when you turn sampling on
or off, you should do so uniformly across all processes, normally at
the same point in the program.  Enabling sampling in only a subset of
the processes would likely produce skewed and misleading results.

And for technical reasons, when sampling is turned off in a threaded
process, interrupts are disabled only for the current thread.  Other
threads continue to receive interrupts, but they don't unwind the call
stack or record samples.  So, another use for this interface is to
protect syscalls that are sensitive to being interrupted with signals.
For example, some Gemini interconnect (GNI) functions called from
inside \verb|gasnet_init()| or \verb|MPI_Init()| on Cray XE systems
will fail if they are interrupted by a signal.  As a workaround, you
could turn sampling off around those functions.

Also, you should use this interface only at the top level for major
phases of your program.  That is, the granularity of turning sampling
on and off should be much larger than the time between samples.
Turning sampling on and off down inside an inner loop will likely
produce skewed and misleading results.

To use this interface, put the above function calls into your program
where you want sampling to start and stop.  Remember, starting and
stopping apply process wide.  For C/C++, include the following header
file from the \HPCToolkit{} \verb|include| directory.

\begin{quote}
\begin{verbatim}
#include <hpctoolkit.h>
\end{verbatim}
\end{quote}

Compile your application with \verb|libhpctoolkit| with \verb|-I| and
\verb|-L| options for the include and library paths.  For example,

\begin{quote}
\begin{verbatim}
gcc -I /path/to/hpctoolkit/include app.c ... \
    -L /path/to/hpctoolkit/lib/hpctoolkit -lhpctoolkit ...
\end{verbatim}
\end{quote}

The \verb|libhpctoolkit| library provides weak symbol no-op definitions
for the start and stop functions.  For dynamically linked programs, be
sure to include \verb|-lhpctoolkit| on the link line (otherwise your
program won't link).  For statically linked programs, \hpclink{} adds
strong symbol definitions for these functions.  So, \verb|-lhpctoolkit|
is not necessary in the static case, but it doesn't hurt.

To run the program, set the \verb|LD_LIBRARY_PATH| environment
variable to include the \HPCToolkit{} \verb|lib/hpctoolkit| directory.
This step is only needed for dynamically linked programs.

\begin{quote}
\begin{verbatim}
export LD_LIBRARY_PATH=/path/to/hpctoolkit/lib/hpctoolkit
\end{verbatim}
\end{quote}

Note that sampling is initially turned on until the program turns it
off.  If you want it initially turned off, then use the \verb|-ds| (or
\verb|--delay-sampling|) option for \hpcrun{} (dynamic) or set the
\verb|HPCRUN_DELAY_SAMPLING| environment variable (static).

\begin{quote}
\begin{tabular}{@{}cl}
(dynamic) & \verb|hpcrun -ds -e event@howoften app arg ...|  \\
(static)  & \verb|export HPCRUN_EVENT_LIST='event@howoften'| \\
& \verb|export HPCRUN_DELAY_SAMPLING=1| \\
& \verb|app arg ...|
\end{tabular}
\end{quote}

% ===========================================================================
\section{Environment Variables for \hpcrun{}}
\label{sec:env-vars}
For most systems, \hpcrun{} requires no special environment variable settings.
There are two situations, however, where \hpcrun{}, to function correctly,
\emph{must} refer to environment variables. These environment variables, and
corresponding situations are:
\begin{description}
  \item{\verb|HPCTOOLKIT|} To function correctly, \hpcrun{} must know
       the location of the \HPCToolkit{} top-level installation directory.
       The \hpcrun{} script uses elements of the installation \verb|lib| and
       \verb|libexec| subdirectories. On most systems, the 
       \hpcrun{} can find the requisite
       components relative to its own location in the file system. 
       However, some parallel job launchers \emph{copy} the
       \hpcrun{} script to a different location as they launch a job. If your
       system does this, you must set the \verb|HPCTOOLKIT|
       environment variable to the location of the \HPCToolkit{} top-level installation directory 
       before launching a job.
\end{description}

{\bf Note to system administrators:} if your system provides a module system for configuring 
software packages, then constructing
a module for \HPCToolkit{} to initialize these environment variables to appropriate settings
would be convenient for users.

\section{Platform-Specific Notes}
\label{sec:platform-specific}

%
% system specific notes for titan, keenland?
%
\subsection{Cray Systems}

The ALPS job launcher used on Cray systems copies 
programs to a special staging area before launching them,
as described in Section~\ref{sec:env-vars}.
Consequently, when using \hpcrun{} to monitor dynamically-linked binaries on Cray systems, you
should add the \verb|HPCTOOLKIT| environment variable to your launch
script.  
Set \verb|HPCTOOLKIT| to the top-level \HPCToolkit{} installation directory
(the directory containing the \verb|bin|, \verb|lib| and
\verb|libexec| subdirectories) and export it to the environment.  
(If launching statically-linked binaries created using \hpclink{}, this step is unnecessary, but harmless.)
Below we show a skeletal job script that sets the \verb|HPCTOOLKIT| environment variable  before monitoring 
a dynamically-linked executable with \hpcrun{}: 

\begin{quote}
\begin{verbatim}
#!/bin/sh
#PBS -l mppwidth=#nodes
#PBS -l walltime=00:30:00
#PBS -V

export HPCTOOLKIT=/path/to/hpctoolkit/install/directory
export CRAY_ROOTFS=DSL

cd $PBS_O_WORKDIR
aprun -n #nodes hpcrun -e event@howoften dynamic-app arg ...
\end{verbatim}
\end{quote}
% $

If \verb|HPCTOOLKIT| is not set, you may see errors such as the
following in your job's error log.

\begin{quote}
\begin{verbatim}
/var/spool/alps/103526/hpcrun: Unable to find HPCTOOLKIT root directory.
Please set HPCTOOLKIT to the install prefix, either in this script,
or in your environment, and try again.
\end{verbatim}
\end{quote}

The problem is that the Cray job launcher copies the \hpcrun{}
script to a directory somewhere below \verb|/var/spool/alps/|\ and runs
it from there.  By moving \hpcrun{} to a different directory, this
breaks \hpcrun{}'s method for finding its own install directory.
The solution is to add \verb|HPCTOOLKIT| to your environment so that
\hpcrun{} can find its install directory.

Your system may have a module installed for \verb|hpctoolkit| with the
correct settings for \verb|PATH|, \verb|HPCTOOLKIT|, etc.  In that case,
the easiest solution is to load the \verb|hpctoolkit| module.  Try
``\verb|module show hpctoolkit|'' to see if it sets \verb|HPCTOOLKIT|.

\subsection{Blue Gene/Q Systems}
Blue Gene Q systems provide the \verb|WALLCLOCK| interval timer, but not the 
POSIX \verb|CPUTIME| and \verb|REALTIME| timers.

The Linux \verb|perf_events| subsystem is unavailable on Blue Gene Q systems. 
One should use the PAPI interface to monitor executions using hardware performance counters. 

\subsection{ARM Systems}
\HPCToolkit{}'s measurement infrastructure depends upon \verb|libunwind| for call stack unwinding on ARM.
On some ARM systems, compilers put DWARF Function Descriptor Entries (FDEs) in the ELF \verb|.debug_frame| segment rather 
than the \verb|.eh_frame| segment. In such cases, \HPCToolkit{} requires a bleeding-edge version of \verb|libunwind| that is not included in 
\HPCToolkit{}'s \verb|hpctoolkit-externals| package. 
\footnote{We are in the midst of deprecating {\tt hpctoolkit-externals} as we move to a spack-based distribution system. While it is used for the
current release, we are no longer maintaining it.}
Contact the \HPCToolkit{} forum if you need a copy of a newer \verb|libunwind|. 
